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Abstract 


This study was premised on the importance of vocabulary in comprehending text. Critical 
findings from both the National Reading Plan (NRP) and the National Reading Technical 
Assistance Center (NRTAC) frame this study, both in the intervention that was implemented and 
in the manner in which outcomes were measured. Using expository passages developed by 
ReadWorks, teachers implemented an ‘Article-A-Day’ with students reading brief expository 
passages. In this particular study, the passages focused on endangered plants and their survival, 
though the full domain of passages available in ReadWorks is extensive and addresses many 
other topics. The primary question was whether this strategy was more effective in supporting 
reading comprehension than when students did not consistently read passages on a daily basis. 
For five days, students read successive passages and, within two days of reading the last passage, 
completed an assessment that was based on a similar (but unfamiliar) passage with key targeted 
words omitted (using a maze format). Significant differences were found in the performances of 
these two groups, with students in the Article-A-Day treatment group performing higher than 
students in the control group. The most important implications of this study include the 
systematic focus on building background knowledge, including vocabulary, in reading 
comprehension instruction in which the active ingredients reflect findings from previous 
research. 
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Introduction 

Nearly 20 years ago, the National Reading Panel (NICHD, 2000) outlined the primary 
areas to address in teaching children to read, using a five component framework: alphabetics 
(phonemic awareness and phonics), fluency, vocabulary, and text comprehension. In this study, 
we address vocabulary. As the subgroup from the NRP noted: “Vocabulary occupies an 
important position in learning to read. As a learner begins to read, reading vocabulary 
encountered in texts is mapped onto the oral vocabulary the learner brings to the task” (p. 4-15). 
This subgroup then cited a number of studies depicting comprehension gains and improvement 
on semantic tasks as the results of vocabulary learning, though the measurement of vocabulary 
can take many different forms. In citing important strategies for teaching vocabulary, they noted 
“that high frequency and multiple, repeated exposures to vocabulary material are important for 
learning gains. In accordance with this finding, a trend was also noted that extended and rich 
instruction of vocabulary (applying words to multiple contexts, etc.) was superior to less 
comprehensive methods” (p. 4-22). Other instructional practices that appeared influential 
included direct instruction, active engagement of students, pre-instruction, and use of rich 
contexts. 

In a follow-up review of research on vocabulary instruction, the National Reading Technical 
Assistance Center (TRTAC, 2010) conducted a research synthesis of 14 studies. They cite three 
main conclusions: 

1. “Higher frequency of exposure to targeted vocabulary words will increase the likelihood 

that young children will understand and remember the meanings of new words and use 
them more frequently” (p. 4). 
2. “Explicit instruction of words and their meanings increases the likelihood that young 


children will understand and remember the meanings of new words” (p. 4). 
3. “Questioning and language engagement enhance students’ word knowledge” (p. 5). 
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With these two reviews, the study was framed on the passages in ReadWorks’ research-based 
reading program Article-A-Day (AAD). AAD uses a 10-minute daily routine focused on three 
critical aspects of reading comprehension: increasing students’ background knowledge, 
vocabulary, and reading stamina. AAD has been shown to help improve students’ reading 
comprehension, and teachers have said that AAD is effective in improving students’ background 
knowledge and vocabulary (Rockman, 2016, 2017). 

Article-A-Day provides students with well-structured expository passages. The passages are 
relatively brief (250-350 words) so students can read them in a single session; they contain clear, 
direct sentences that provide content on a topic. The underlying assumptions behind the reading 
program include frequent (daily) reading of passages with a similar discourse structure having 
key vocabulary presented in context and active application of the information in a context. 

Therefore, the goal of this pilot was to learn if AAD improved students' reading 
comprehension by giving them new background knowledge that they applied to understanding 
new texts with a similar structure. Specifically, our research question was whether a significant 
positive effect occurred from students completing AAD by reading a set of five topically 
connected articles as measured by maze vocabulary performance on a new text related to the 
same topic. To conduct a study of the effects from using AAD as an intervention, a dependent 
measure was designed to reflect a new passage that represented the same discourse structure with 
key vocabulary (on a new but related topic). This outcome assessment was developed using a 
passage focused on the Alula plant, which was a new article that students had not read previously 
but reflected the same text structure and similar topic (on endangered plants) as the previous 
AAD. The idea is that the vocabulary crosses among the articles and therefore students learn it 


more deeply. This type of study on expository text using a maze assessment has been 
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successfully deployed previously to reflect an emphasis on concepts and vocabulary present in 
subject areas (McCoy, Twyman, & Tindal, 2006; Twyman, McCleery, & Tindal, 2006). 


Methods 


In this section, we describe the participants, the treatment materials, the outcome 
measure, and the analyses that we conducted. The primary purpose of the study was to determine 
the (experimental) effects of using AAD, but we also were interested in creating and analyzing a 
measure for documenting these outcomes: a maze measure. 


Participants 


Volunteer teachers were recruited from the database of teachers registered on the 
ReadWorks digital website. The target goal of number of participants was 100 teachers in each 
of the control and treatment groups. This number was determined to be sufficiently large to draw 
initial conclusions from this pilot study. 

First, the database of registered teachers was divided into two geographical areas: 
teachers in New York City (NYC) and teachers not in NYC but still in the U.S. The reason for 
this division was that a funder for the project, the Brooke Astor Fund for NYC Education, 
focuses specifically on supporting programs and activities that improve the quality of education 
in NYC. Therefore, the results of the experiment needed to be analyzed for students in NYC and 
students overall. 

Within these two geographical areas, criteria were set to determine the teachers who 
would receive the first email request (sent between April 11-30, 2019) asking them to voluntarily 
participate in either the control or treatment group. A second round of email requests (sent May 
14, 2019) was deemed necessary to recruit more participants. Specifically, the number of NYC 
participants who volunteered after the first request was much smaller than the target goal, and, in 


both NYC and the national geographic areas, the control group participation far exceeded the 
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treatment group participation. In the second round of requests, some of the criteria were 
eliminated to reach a larger number of possible participants. These are indicated below in 
parentheses. 

Criteria for NYC Treatment Group: (a) NYC DOE school, (b) Grade 4 class, (c) digital 
class that had completed digital assignments (not a criterion for the second round), (d) four or 
more AAD uses, (e) no prior exposure to the Endangered Plants AAD set or any of the individual 
articles within the set (not a criterion for the second round). 

Criteria for NYC Control Group: (a) NYC DOE school, (b) Grade 4 class, (b) digital 
class that had completed digital assignments (not a criterion for the second round), (c) no use (or 
single use) of an AAD set, (e) no exposure to the Endangered Plants AAD set or any of the 
individual articles within the set (not a criterion for the second round). 

The same criteria were deployed for the national control and treament groups as was used 
with the NYC control and treatment groups with a few exceptions. The control group (a) was of 
sufficient size to not require a second round of requests, (b) had identified their school name in 
the ‘school’ field item within the ReadWorks, and (c) had not received outreach for a different 
ReadWorks experiment. Furthermore, the treatment group (a) had identified their school name in 
the ‘school’ field item within the ReadWorks, (b) had used AAD with 10 or more passages, and 
(c) had not received outreach for a different ReadWorks experiment. 


Treatment Materials 


The AAD program uses weekly sets of articles that are topically connected, such as a set 
of articles on storms or physical science. For this study, we used the AAD set about endangered 
plants because this is a topic that aligns with common curricular topics for the spring in fourth 
grade. We hoped this would make participating in the study less disruptive for teachers, thus 


increase the number who volunteered. While the AAD sets often have more than five articles to 
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allow for teacher and/or student choice, we limited the endangered plants set to only five articles 
for this pilot to ensure that students read similar content prior to taking the maze. The five 
Article-A-Day passages that were read by students in the treatment group included: (a) “The 
Most Stinky Flower on Earth,” (b) “Survivor Trees,” (c) “The Upside Down Tree of Life,” (d) 
“Why Are Some Plants Endangered,” and (e) “Steps You Can Take to Save Endangered Plants.” 
Each of the passages was approximately 300 words in length and designed to be read in a single 
session. Teachers generally use the passages to supplement instruction. 


Study Design 


The study involved a control and treatment group. In the control group, teachers had their 
students complete the 15-question maze. Students had not read anything in preparation for the 
maze. Teachers were encouraged but not required to limit students to 10 minutes to take the 
assessment. In the treatment group, teachers were asked to implement the following protocol: (a) 
using the endangered plants AAD set, guide students through the AAD routine by reading one 
article a day for five days; (b) within 48 hours, ideally but no more than one week after this AAD 
routine was complete, have students complete the 15-question maze. Teachers were also 
encouraged, but not required, to limit students to 10 minutes to take the assessment. For the 
control group, we confirmed (within their use metrics on our digital platform) that they had not 
assigned the endangered plant AAD set, thus that they had followed the protocol. For the 
treatment group, we only included data in our analysis from teachers whose use metrics showed 
that they had both assigned the endangered plants AAD set and had their students complete the 
maze. 


Outcome Measure 


We developed a maze reading comprehension measure, a commonly used formative 


assessment to measure reading comprehension that has been established as both reliable and 
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valid for assessing reading comprehension progress (Fuchs & Fuchs, 2002; Marcotte & Hintze, 
2009; Pierce, McMaster, & Deno, 2010; Shin, Deno, & Espin, 2000; Shin & McMaster, 2019). 
The maze generally uses a standardized cloze format in which students read a text with at least 
300 words that, after the first sentence, has every seventh word removed and replaced with three 
choices: the correct word and two incorrect words that served as distractors (Fuchs & Fuchs, 
2011, 1992). This format, however, has not always been strictly followed; for example, in some 
uses, the correct word and distractors are selected purposely to increase test difficulty (Shinn & 
Shinn, 2002). 

Therefore, we adapted the maze format to try to test students’ application of background 
knowledge and vocabulary to a new text. Instead of removing every seventh word, we removed 
keywords that would signal background knowledge about the AAD topic, as outlined in Liu, 
Sundstrom-Hebert, Ketterlin-Geller, and Tindal (2008) who used both classical and item 
response theory (IRT) to document reliability and validity data for this assessment type. 
Specifically, our maze reflected a vocabulary assessment that was contextual (Pearson, Hiebert, 
& Kamil, 2007); furthermore, the maze options (both correct and incorrect) were designed to be 
meaningful in sentences as both content-related and the same part of speech as the eliminated 
word (Parker, Hasbrouck, & Tindal, 1992). Finally, only three options were given, as per the 
meta-analysis conducted by Rodriguez (2005). In the construction of the options, we used 
Hiebert’s (2005, 2006) semantic associations as a critical criterion in selecting vocabulary words 
for the maze (Pearson et al., 2007), though a second criterion was instructional presence using 
AAD within the teaching-learning cycle, given the treatment of previous AAD articles on 


endangered plants. 
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Our 15-item maze was about a specific endangered plant, the Alula, that had not been 
mentioned in any of the articles in the AAD endangered plants set. The passage was comparable 
in length and Lexile and formatted similarly to the articles in the endangered plants AAD set. 
The words we selected to omit in the maze were important words in the vocabulary of the study 
of endangered plants (e.g., damaged, invasive). In this way, the maze required students to use 
their background knowledge about endangered plants to choose the correct word as they read 
about an unfamiliar endangered plant. 


Analyses 


We have summarized participation data using traditional descriptive statistics to 
document participants and performance. Importantly, to ensure the treatment and control groups 
were implemented with fidelity, we analyzed two questions about their use of AAD and then 
removed those who confounded the treatment. Then we analyzed the two groups using multiple 
regression, eventually using three conditional models to control for time (both unconstrained and 
limited to one hour). Finally, because we were interested in analyzing the maze measure to 
determine its suitability for use in evaluating AAD, we conducted extensive analyses of the 
outcome measure, including an option analysis and its reliability. 


Results 


We summarize the results first with a preliminary exploration of the data set summarizing 
participation (of teachers and students) using descriptive statistics and graphic displays of student 
groups within teachers. We then summarize time spent completing the maze, again presenting 
graphs to show the distribution. Finally, we map teacher participation from the various sites in 
the country. The most important analysis is then presented in which we compare the treatment 
and control groups on the maze measure, first graphically displaying distributions and then 


analyzing statistical significance using multiple regression. Finally, we present an option analysis 
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of each item in which the proportion of students is displayed on the y-axis for each option with 


their total score displayed on the x-axis. The last analyses summarize the reliability of the maze. 


Descriptive Statistics 


The initial data analyses addressed the sample of teachers and students. Considerable 
variance was present in the number of students within each teacher’s classroom. This distribution 
was bi-modal and generally reflected a group of teachers with under 10 students and another 
group with around 20-25 students. A few teachers had between 40 and 50 students. These 
distributions were generally similar for both Treatment (A) and Control (B) teachers. 

Students from both the treatment and control groups took approximately the same amount 
of time (7 minutes); considerable variation, however, existed in the time taken to complete the 
maze (with a standard deviation of nearly 5 minutes). 

The vast majority of teachers had not previously assigned the content or previously 
projected/printed the content. More Treatment than Control teachers had previously assigned the 
content; in contrast, more Control than Treatment teachers had previously projected or printed 


the content. 


Table 1 


Use of Article-A-Day by Intervention and Control Teachers 


Exposure Treatment Control 
Did not previously assign content 495 LET 
Previously assigned content 70 20 
Exposure Treatment Control 
Did not previously project or print content 559 753 
Previously projected or printed content 6 44 


In both cases of exposure, we removed any teachers from the comparative analysis if they had 


assigned, projected, or printed the content. 
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Figure 1. Distribution of number of students within teachers. 
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Figure 2. Distribution of number of students within teachers by treatment and control group. 
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Table 2 


Time Taken (limited < 60 mins) 


Group Mean SD 

A 6.72 mins 4.44 

B 6.89 mins 3.50 
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Figure 3. Amount of time taken to complete the maze. 


Map of Participation 


Using longitude and latitude information in the data file, a geo-spatial map was 
developed to show the sites participating in the study. The greatest concentration of sites were 
from the mid-west and east with California and Arizona representing the majority of the western 


sites (and single sites in Oregon, Washington, and Alaska). 
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Figure 4. Geo-spatial distribution of participating sites. 


Comparison of Treatment versus Control Groups 


The distributions of performance for both the treatment and control groups overlapped 
considerably. When placed on the same plot, the Treatment distribution (green plot) was slightly 
higher than the Control distribution (orange plot). The vertical line marks the mean with 
Treatment students slightly higher (more than one half point). When viewed as stacked 
distributions (both blue), Treatment students showed a slightly more narrow distribution (fewer 
students in the lower end and a tighter group of students in the higher end) than Control students 
Following the two distribution graphs, we display a box and whisker box plot. The top of each 
box (Control and Treatment) represents the score associated with the 75th percentile rank (PR); 


the middle of the box reflecting the score associated with the 50th PR; the bottom of the box 
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reflecting the 25th PR. Outlier scores are above and below (the line extending above to the 90th 
PR and below to the 10th PR) and individuals with data points. The primary interpretation from 
these plots is that students in the Control group were slightly lower, particularly from the 50th 
PR and below. 

Finally, a bar chart presents the performance of students in New York City who 
participated (all of them in the Treatment Group). Similar to the larger distributions, these 
students performed in the upper score range; only a couple scored in the low end with everyone 


above 10 (of the 15 words) and most between 12-14 words correct. 
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Figure 5. Distribution of scores by treatment and control groups. 
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Figure 6. Distribution of scores by treatment and control groups. 


Evaluating ReadWorks AAD with a Maze Measure 


15 

= 10 

Oo 

(ep) 

& 4 

(eo) 

- 5 t 
a e 
eo 

O 2 
Control Treatment 
Group 


Figure 7. Box and whiskers plot of scores on maze by treatment and control groups. 
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Figure 8. Bar chart of maze scores for the New York city participants. 


Given these slight differences between the Treatment Group (A) and the Control Group 
(B), the question focused on whether it was significant. The regression analysis indicated that, 


when using the Control Group as a reference: (a) the intercept was about 11 words correct and 
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(b) the Treatment Group was significantly higher on the maze than the Control Group (with more 
than an additional half point gained for each point attained by students in the Control group). The 
probability of being wrong in this conclusion is less than .001, but the variance accounted for 
represents a relatively small amount (e.g., R2 was low with a value of .012). 

Table 3 


Regression Table Depicting Significance of Treatment Relative to Control 


() 
(Intercept) 11.430 *** 
(0.107) 
Treatment 0.646 *** 
(0.169) 
N 1222 
R2 0.012 
logLik -3029.694 


AIC 6065 .388 


EE Re OO LS SF p< 
0.01: *p-<005, 


In the next set of analyses (with column 1 repeating the analysis above), time was taken 
into account as a covariate in columns 2 (time unconstrained) and 3 (time limited to one hour or 
less). We measured time as the difference between the last opened and the submitted response. 
Although the model in column 3 indicated a significant effect from the treatment, no effect was 
present when time was taken into account; the amount of variance explained (R2) did not 


change. 
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Table 4 


Regression Table Depicting Significance of Treatment (Group A) with Three Models 


(1) (2) (3) 
(Intercept) 11.430 11.650 11.650 
ok ok 2k ok ok 2k 2k ok 2k 
(0.107) (0.181) (0.181) 
groupA 0.646 0.641 0.641 
2k 2k 2k ok ok 2k 2k 2k 2k 
(0.169) (0.169) (0.169) 
time_taken -0.032 -0.032 
(0.021) (0.021) 
N 1222 1222 1222 
R2 0.012 0.014 0.014 
logLik - - - 
3029.6 3028.552 3028.552 
94 
AIC 6065.3 6065.104 6065.104 
88 


* p< 0.001; ** p< 0.01; *p <0.05. 


In the final set of analyses, each item is analyzed for its functioning: (a) in relation to the 


distractors and (b) in terms of reliability (consistency). Each plot shows the three options with 
the correct answer listed first and the two incorrect options listed subsequently with the colored 
lines being a smoothed fit and the gray line reflecting the actual values. The plot displays the 


relation between the option being selected (the proportion of students selecting it on the y axis) 


and their performance on the maze on the x axis (with lower scores on the left and higher scores 


on the right). Consistently, a relation exists with proportionately more students selecting the 


correct option as they perform with higher total scores. The displays also show the relative lack 
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of consistent distraction for the two incorrect options. Finally, the plots reveal a ceiling effect; 


the maze was relatively easy. 


Item Option Analysis 
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Figure 9. Distribution of proportion correct on total score by option: Item 0. 
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Figure 10. Distribution of proportion correct on total score by option: Item 1. 
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Figure 11. Distribution of proportion correct on total score by option: Item 2. 
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Figure 12. Distribution of proportion correct on total score by option: Item 3. 
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Figure 13. Distribution of proportion correct on total score by option: Item 4. 
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Figure 14. Distribution of proportion correct on total score by option: Item 5. 
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Figure 15. Distribution of proportion correct on total score by option: Item 6. 
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Figure 16. Distribution of proportion correct on total score by option: Item 7. 
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Figure 17. Distribution of proportion correct on total score by option: Item &. 
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Figure 18. Distribution of proportion correct on total score by option: Item 9. 
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Figure 19. Distribution of proportion correct on total score by option: Item 10. 
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Figure 20. Distribution of proportion correct on total score by option: Item 11. 
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Figure 21. Distribution of proportion correct on total score by option: Item 12. 
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Figure 22. Distribution of proportion correct on total score by option: Item 13. 
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Figure 23. Distribution of proportion correct on total score by option: Item 14. 


In the tables below, overall reliability of the maze is first presented, and then reliability 
(consistency) of each item is listed in raw score values, standardized values, and with the 
standard error. The maze assessment overall is reasonably reliable overall (.76) and with all 
items (ranging in the same values). 

The final tables present item statistics with the reliability presented as raw score values, 


standard score values, when being answered correctly, and with the item being dropped; finally, 
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the average (mean) and standard deviation (sd) are presented. The important interpretation from 
this table is the high mean for all the items except for item 5. 
Reliability Analysis 


Table 5 


Overall Internal Consistency for Maze 


estimate lower upper 
0.76 0.74 0.78 


Table 6 
Reliability with Item Removed 


raw_alpha std.alpha alpha se 


sO 0.74 0.75 0.01 
sl 0.74 0.75 0.01 
s2 0.75 0.76 0.01 
s3 0.74 0.75 0.01 
s4 0.75 0.76 0.01 
s5 0.76 0.77 0.01 
s6 0.74 0.75 0.01 
s7 0.74 0.75 0.01 
s8 0.75 0.76 0.01 
s9 0.74 0.75 0.01 
s10 0.74 0.75 0.01 
sll 0.77 0.77 0.01 
s12 0.74 0.75 0.01 
s13 0.74 0.75 0.01 


sl4 0.75 0.76 0.01 
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Table 7 


Item Statistics 


n rawr std. r.cor r.drop mean sd 

sO 1220 053 053 047 O41 0.79 O41 
sl 1218 O57 057 053 046 0.82 0.39 
s2 1221 040 045 0.38 0.32 0.95 0.23 
s3. 1215 053 054 050 043 0.84 0.36 
s4 1217 040 041 0.33 0.28 0.85 0.35 
ss 1216 0.39 035 0.25 0.23 035 048 
s6 1215 052 052 048 O41 0.83 0.37 
s7? 1215 054 056 052 045 088 0.32 
s8 1215 049 048 042 0.36 0.76 042 
s9 1210 054 051 046 040 0.70 046 
slO0 1208 054 051 046 O41 072 O45 
sll 1207 0.31 029 0.18 0.16 0.75 043 
sl2 1204 058 058 055 048 0.82 0.38 
s13. 1202 053 053 048 042 081 0.39 
sl4. 1201 042 045 038 0.33 O91 0.29 
Discussion 


This study addressed the effects from using AAD to improve student comprehension of 
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brief expository passages. After five days, significant differences appeared between students who 


consistently read the passages and those who had not read them. This difference was both 


statistically and clinically significant: With a .01 chance of being incorrect in the claim, students 


in the treatment group improved over one point on the maze for every point achieved by the 


control group students. This finding may be a function of several factors. 


the sample size was large which makes such results easier to achieve, providing a high level of 
power. Furthermore, this finding may need to be qualified by the difference in sample sizes for 


the treatment and control groups, making the analysis unbalanced. 


Certainly, the properties of the statistical analyses need to be considered. For example, 


Evaluating ReadWorks AAD with a Maze Measure 25 


Nevertheless, the results are encouraging for a pilot study and are consistent with the 
previous research on reading comprehension by national groups. Both the NRP and NRTAC 
emphasize the importance of vocabulary in teaching and learning students’ comprehension. We 
emphasized vocabulary in this study because of both the treatment and the measurement of 
outcomes. AAD consists of well-controlled passages of approximately 300 words that can be 
analyzed in a conceptual framework with key vocabulary. In the five AADs used as part of the 
treatment and the maze measure used to document outcomes, the key concepts addressed (a) 
characteristics of a plant (mostly adjectives with a few nouns), (b) the threats to their survival 
(mostly verbs with a few adjectives), and (c) sources of the threats (mostly nouns). Thus, when 
students are reading, they may be organizing the content into both lexical/semantic and 
grammatical categories. The AADs also reflect components of effective vocabulary instruction 
summarized by the NRP and NRTAC: more opportunities to be actively engaged in reading, 
repeated exposure to targeted vocabulary, explicit instruction supplemented by teachers, and 
follow-up questioning provided by teachers (though neither of these last two strategies were 
monitored). 

Given that the purpose of the study was to both investigate the effects of Article-A-Day 
and document the use of a maze assessment, this study focused on vocabulary as a key 
component of both the treatment and the outcome. Although the treatment effectiveness was 
supported (with students in the AAD [treatment] group outperforming students in the control 
group), two issues appeared with the maze as an outcome assessment. First, the reliability was 
moderate, both with the overall measure and with items being individually removed (Tables 5 
and 6). Second, the distributions of the individual items plotted with proportion correct for each 


option as a function of the total score displayed an appropriate curve for the correct option but 
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flat curves for both distractors. In essence, the distractors did not function appropriately for 
students with different total scores. This is likely a contributor to the relatively high mean score 
for each item (other than item 5). It is therefore important that construction of the maze needs to 
take into account both what key vocabulary words are omitted, but also what words are used in 
their place (Liu, Sundstrom-Hebert, Ketterlin-Geller, and Tindal, 2008). Given these two 
qualifications of the maze, support for the AAD appears well documented, particularly given the 
lack of control over actual instructional strategies in the classrooms and the brief duration of the 


treatment. 
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